As mentioned above the hookup positioning does no harm physically to the driver. It does alter the system response, though. That is what you are experiencing. To what extent depends upon a lot of factors. The change in polarity introduces a different wave phasing. How these phase angles mix at the crossover point is what matters. The combination of driver phasing has the effect of cancelling output around the xover point (a suck out) or adding output (a hump) in the extremes. The designer will tweak the final xover design compensating for the phase characteristics to get a more flat response around the xover point. Are you using an ALTEC xover designed for the 604? Wondering if you had the connection out of spec originally and the change pulled it back into spec?